Embodiments of the present invention generally relate to computer systems, and more specifically to techniques for asynchronous file replication and migration in a storage network.
File systems are most commonly stored on random access storage devices (like spinning magnetic disks). Traditionally these disks were directly attached to a single computer, via a disk controller. Many years ago it was the case the files on a disk could only be accessed by programs that were running on the computer to which the disks were attached.
The advent of standardized remote file access protocols (e.g., the Network File System and the Common Internet File System) have made it possible for clients to access files on other computers. The advent of Storage Area Networking technologies (such as fibre channel and iSCSI) has made it possible for a single storage device (e.g., a disk) to be accessed directly and efficiently by multiple computers. In this new regime, client software is no longer aware of what device their files are actually stored on, and it becomes possible to move files from one storage device to another.
In order to provide scalability in the size and bandwidth of a distributed file system, it is necessary to spread the managed files across multiple storage nodes. There are many reasons to move files from one storage volume to another.
Data that is frequently referenced can be delivered more quickly if clients have multiple copies from which they can read. Data that is critical can be replicated so that copies will continue being available even if there is a failure on its primary storage device. If multiple copies exist of a file, and one of those copies is updated, the updates must be propagated to all of the other copies. Data that is no longer popular can be moved from more expensive primary storage to less expensive secondary storage, freeing up the primary storage for more valuable uses. If many heavily used files are stored on a single device, a long line may form. If those files can be spread across multiple devices, service will be faster. As files grow in size and number, the device on which they were stored may no longer be large enough to hold them, and it will be necessary to move (some or all of) them to a larger device.
In response to these issues, system managers are often called upon to move files from one device to another. This is often done manually, and usually involves taking the affected devices out of service (so that the files are not available while they are being moved). Manual transfers are slow and labor intensive. Taking file systems out of service to perform such operations disrupts normal operations. As a result, these file movements are not performed as soon or as often as they should be.